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Objective evaluation 
of the subjective 


DEFINING ANALYST PRODUCTIVITY 

r (b)(3)(c) 


The Directorate of Intelligence has centered its attention on improving 
organizational productivity by emphasizing the increased flow of high-quality 
memorandums. While the goal is straightforward the measurement of suc- 
cess at least in terms of quality — is not. The evaluation of product quality is 
subjective; more often than not the determination is made within the 
organization rather than by intended consumers. Moreover, it is difficult to 
weigh the relative contributions of various types of intelligence production. 
The debate between the respective merits of current reporting and in-d<§pth 
analysis has persisted for years. It is similarly difficult to address the relative ~ i 
contributions of written monographs, oral briefings and, increasingly, video- 
taped presentations. 

The problem of improving DDI production is exacerbated by the relative 
inexperience of DDI analysts and first-line managers. Our business in many 
ways requires a complex and often subtle set of skills. Specifically, accom- 
plished intelligence officers have developed a unique set of research, analysis, 
and expositional talents along with a clear appreciation of policymaker needs. 
These talents have traditionally been developed and honed by extensive on- 
the-job experience; the trick is to accelerate this process. 

A number of steps have been taken to address the issue of improved DDI 
productivity. To better ensure a consistently high-quality product, DDI 
intelligence production receives an in-depth high-level review. Concomitantly 
DDI managers have been encouraged to develop a rapport with their major 
consumers as a means of both gaining an understanding of intelligence 
community needs and nurturing honest criticism of Agency production. A 
number of management and analyst training programs have been initiated to 
increase the skill base at the working level. The subjective nature of the 
product, however, has made and will continue to make productivity improve- 
ments difficult. 

Intelligence Directorate Product 

A full discussion of the DDI “product” could easily fill several pages. 
Beauty lies in the eyes of the beholder, and the product, by design, takes many 
forms— daily and weekly periodicals gisting key events for high-level recipi- 
ents, the dissemination of in-depth monographs on specific topics of interest, 
individual briefings, and informal dialogue with working level colleagues in 
the intelligence consuming agencies. It is not my intent to provide a definitive 
description of DDI production but rather to establish a baseline definition 
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which sets the stage for the ensuing analysis of productivity. For purposes of 
discussion therefore, I would assert that: 

The Intelligence Directorate exists to analyze and report to 
high-level policymakers on foreign political, military, and 
economic trends and activities. The quality of DDI products is 
determined by their relevance, prescience, analytical founda- 
tion, and the cogency and incisiveness of exposition. 

Criteria for Measuring Productivity 

Judgments regarding analyst productivity have differed both among and 
within DDI offices. In part, these differences stem from varying office 
perspectives on the nature of their contribution. Some place great stock in 
preparing analysis for the daily periodicals while others put great store by the 
development of a cadre of experts capable of providing sophisticated analysis 
upon request. 

Line managers, in turn, have traditionally drawn from personal exfieri- 
ence when guiding analyst development and deciding upon criteria for J 
analysts’ performance appraisal reports (PARs). As a result, analysts have 
differing perceptions regarding the nature and quality of their individual 
contribution and how it can be improved. Reviewers of analyst files — 
particularly from other offices— must often guess at the ability of analysts to 
function within the reviewers’ respective environments. DDI management 
courses, while going into detail on the performance evaluation process, offer 
less in the way of clearly delineating those criteria that distinguish analyst 
productivity. 

I believe clear guidelines addressing analyst productivity would be a 
major step toward alleviating these problems, benefiting analysts and man- 
agers alike. The former would gain from a set of performance criteria which 
accurately capture their expected contribution and thereby give them a clear 
determination of their relative strengths and weaknesses. The latter would be 
forced to validate their judgments of analyst performance in terms of a 
standard definition of productivity. 

The following five criteria, allowing for variations in wording to reflect 
individual circumstances, capture what I think is the essence of analyst 
productivity within the DDI. While the following exposition is pertinent for 
political economists, the PAR criteria could easily be altered to take other 
disciplines into account. The criteria are: 

• The research of political/economic phenomena within assigned 
areas of responsibility. 

• The analysis of those phenomena. 

• Written and oral exposition. 

• Awareness of policy interest in assigned areas of responsibility. 

• Participation in the coordination and review process. 
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$ While research, analysis, exposition, and policy relevance can he discussed 
independently, there is a strong synergistic relationship among the four. A 
good understanding of policy needs focuses information collection and the 
thrust of the ensuing analytical effort. Good research and analytical skills lend 
themselves to a cogent exposition, while good expositional talents, in turn, help 
focus the analytical thrust of the paper. This synergism is perhaps the main 
reason behind the difficulty in delineating analyst productivity; it is what 
makes our business an art rather than a science. Nonetheless, by paying 
specific attention to the factors described below both analysts and managers 
can better identify strengths and weaknesses and, in so doing, improve the 
quality of future work. 

Research. Research skills, in their broadest sense, are evidenced by the 
breadth and richness of the material that individual intelligence analysts are 
able to bring together as part of the analytical effort. Specifically, good 
research skills bring to the analyst’s desk policy-relevant information which 
might otherwise have gone unnoticed. The analytical process, in turn, 
determines the direction and scope of additional research. In other wPfds, the 
parameters of the analytical effort are often determined by the initial base o 3 f 
information while the outcome of analysis is influenced by the research tools 
utilized. To this end good research minimizes the potential for errors of 
omission or commission which can debase the credibility of otherwise 
excellent analysis. 

In my estimation, an analyst s research capabilities are determined by 
his/her: 

• Imaginative use of intelligence community collection and infor- 
mation retrieval systems, including the development of fruitful 

- dialogues with intelligence collectors. 

• Use of automated data processing capabilities and the resources 
of programming specialists to optimize data management and 
analysis. 

• Ability to adjust the duration and sophistication of research 
endeavors in line with topic complexity and system-imposed 
time constraints. 

• Proper filing of intellig ence data and imaginative use, as 
appropriate, (b)(3)(c) 

• Active review of relevant analysis conducted elsewhere within 
the government or within the private sector. 

Analysis. The analytical process is understandably the most difficult to 
evaluate in an objective manner. The diversity of subject matter and product 
does not lend itself to broad generalizations and the analytical process is 
appropriately different for each line analyst. Nonetheless there are character- 
istics that affect the quality of intelligence analysis across the DDI. These 
include the analyst’s: 

• Ability to work independently or as part of a group. 
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$ * Creativity to do analysis based on incomplete data. 

• Vision to look forward in time. 

• Character to make judgments when all the facts are not known 
or to state when there is insufficient information to support a 
judgment. 

• Acceptance of the need to incorporate divergent opinions into 
individual analysis and the willingness to seek alternative view- 
points from both within and without the US intelligence 
community. 

• Ability to tackle complex issues in depth as well as do short-fuse 
policy support. 

Exposition. Analysts often underestimate the importance of clear exposi- 
tion to the quality of the final product and the positive role they can play in 
this process; many find the review process onerous and counterproductive. 
Rather than working diligently to improve their writing skills so * to 
minimize editorial changes, they accept extensive review as a* necessary cost. 
Similarly, analysts and managers often pay insufficient attention to strength- 
ening briefing skills as a means of improving the impact of the analysis on the 
policymaker. The following criteria, while targeted toward written exposition, 
can be easily amended for those whose product is presented orally. These skills 
include: 

• Cogency and clarity of exposition. 

• Ability to meet short deadlines arid length limitations. 

• Imaginative use of charts and graphs. 

• Policy focus of the presentation. 

Policy Interest. If our product is to have any impact or, for that matter, to 
be read at all, it must be pertinent to policymaker interest. This holds true 
both for the general subject and the focus of our analysis on the subject in 
question. Too often analysts, and for that matter first-line supervisors, relegate 
policymaker contacts to upper management; research plans, the timing of 
papers, and the focus of analysis are determined with only a general sense of 
the need downtown. There is, in fact, a major role that can and should be 
played at the working level. Dialogue with potential consumers serves to locate 
specific policymaker interests, to identify gaps in his knowledge base, and to 
suggest the appropriate timing of the product. This same dialogue will serve to 
elicit honest criticism and comment that serve to improve future intelligence 
production. To this end, analysts should be judged, in part, on their: 

• Development of working level contacts within the intelligence 

community and with government agencies that make and carry 

out policy. 

• Review of open press coverage — newspapers and periodicals — 

discussing policy interest on assigned areas of responsibility. 
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* * Coor( hnation and Review. An analyst’s productivity goes beyond his 

persona] research, analysis, and drafting. Analysts, and often first-line manag- 
ers as we 1 tend to lose sight of the corporate nature of our business. The prod- 
uct should reflect the full knowledge of the Directorate. Our readership 
deserves no less while our masthead indicates that this, in fact, is the case. 

The coordination and review process can substantially improve quality by' 
incorporating additional facts, judgments, and perspective and by improving 
the paper s exposition. An analyst’s productivity is significantly enhanced by 
his positive and creative participation in this process. Many analysts however 
tend to be very parochial about their work, making the review and coordina- 
tion process contentious and laborious. The resulting frustrations, over time 
often lead to an equally undesirable throwing up of the hands and increasing 
analyst detachment from-and thereby a declining commitment to-his. 
paper as it winds its way through the review process. 

The full participation of analysts in coordinating papers authored by 
others is equally important. Too often analysts regard coordination responsibil- 
ihes as a drag on their own efforts. If the paper does not threaten their own 

, tu 5,: , CO ? r r nat !°" tends to be su P er Scial; gross errors or omissions are 
highlighted, but differing perspectives or the availability of additional infor- 
mation are not communicated. The coordination of papers touching on turf, in 
contrast, is often dominated by confrontation, nitpicking, and other obstruc- 
tionist tactics; all too rarely can competing analysts sublimate their respective 
egos to the greater end of a fully integrated product. Those analysts who do 
pay an active and positive role in the coordination process often have a 
significant impact on the quality of the Directorate’s product; their contribu- 
tion in this area, however, often goes unrecognized by line managers. 

Role of the Line Manager 

The role of the line manager is critical in identifying analyst strengths and 
weaknesses and otherwise nurturing analyst improvement. To play a credible 
role in enhancing productivity, the line manager must have a clear under- 
standing both of the factors his office considers most pertinent to measuring 
productivity and of how his analysts stack up against these criteria He must 
subsequently impart this information to his staff; specifically, he must make 
sure his analysts understand the productivity criteria and, accordingly, their 
strengths and weaknesses. The PAR system provides a vehicle for such 
discussions but does not mandate them. It is significant, in my opinion, that 
managers are forced to quantify analyst productivity as part of competitive 
ranking exercises but need not relay this information to the analysts While I 
would agree a manager-analyst discussion of the competitive ranking criteria 
could prove unduly confrontational and counterproductive, a clear and honest 
discussion of analyst productivity should not be left off the table. 

Improving analyst productivity need not be an onerous and confronta- 
t !°" al pr ° cess - Strong analysts seek guidance on how to further improve their 
s ills while weaker ones benefit from constructive and specific guidance from 
the line supervisor. The process fails when either the analyst is not made fully 
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awafe of his relative strengths and deficiencies or — if informed — is not given 
concrete guidance on ways to improve his performance. Such breakdowns in 
communication often can be traced to the inability of line managers and 
analysts to clearly understand the nature of the productive process. This 
discussion offers one man’s views on how to ameliorate this problem; other 
guidelines may be equally appropriate. 
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